IES RV Institute of 
Ay Education Sciences 


¢ WHAT WORKS 
eMINTS Comprehensive Program CLEARINGHOUSE™ 


yal AIAN) 


NCEE WWC 2020-011 
U.S. DEPARTMENT OF EDUCATION 


A Publication of the National Center for Education Evaluation at IES 


Research has shown that teacher effectiveness is the most high-quality, inquiry-based learning, in which students develop 
important school-based factor that influences student outcomes, understanding and knowledge of content matter by engaging in 
including student achievement. Studies have also shown that meaningful investigations that require reasoning, judgement, and 
there is substantial variation in teacher effectiveness to improve decision making. The intervention can provide support to teachers 
student outcomes.' The strong influence of teachers on student in any subject area, including math, literacy, and science.” 


outcomes, as well as the variation in teacher effectiveness, 

has led to the creation of programs designed to help teachers 
improve their effectiveness in the classroom. The eMINTS 
Comprehensive Program aims to help teachers improve their 
practice and the outcomes of their students by offering structured 
professional development, coaching, and support for integrating 
technology into the classroom. The program’s goals include 
supporting teachers in using classroom technology to implement 


This What Works Clearinghouse (WWC) report, part of the 
WWC’s Teacher Excellence topic area, explores the effects of 
the eMINTS Comprehensive Program, referred to as eMINTS in 
this report, on student achievement. The WWC identified eight 
studies of eMINTS. Two of these studies meet WWC standards. 
The evidence presented in this report is from two studies of the 
impact of eMINTS on students in grades 4, 5, 7, and 8. 


What Happens to Students When Their Teachers Participate in eMINTS?3 


The evidence indicates that implementing eMINTS: effectiveness rating, the improvement index, and the number 
of studies and students that contribute to the findings. The 
improvement index is a measure of the intervention’s effect 
on an outcome. It can be interpreted as the expected change 


¢ May increase general mathematics achievement 


¢ May result in little or no change in general literacy 


achievement in percentile rank for an average comparison group stu- 
Findings on eMINTS from two studies that meet WWC dent if that student had been placed in the classroom of an 
standards are shown in Table |. The table reports an eMINTS teacher. 


Table 1. Summary of findings on eMINTS from studies that meet WWC standards 


Study Findings Evidence meeting WWC standards (version 4.0) 
Improvement index 
Outcome domain Effectiveness rating (percentile points) Number of studies Number of students 
General mathematics achievement Potentially positive effects +2 2 3,096 
General literacy achievement No discernible effects 0 2 3,245 


Note: The improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the 
intervention. For example, an improvement index of +2 means that the expected percentile rank of the average comparison group student would increase by 2 points if the 
student were placed in an eMINTS classroom. The improvement index values are generated by averaging findings from the outcome analyses that meet WWC standards, 

as reported by Brandt et al. (2013) and Martin et al. (2009). A positive improvement index does not necessarily mean the estimated effect is statistically significant. General 
mathematics and literacy achievement outcomes reported in these studies are scores from the Missouri Assessment Program standardized tests in each subject. The effects 

of eMINTS are not known for other outcomes within the Teacher Excellence topic area protocol, including general science achievement, general social studies achievement, 
general achievement, English language proficiency, staying in school, progression in school, completing school, student social interaction, observed individual behavior, student 
emotional status, student engagement in school, instructional practice, teacher attendance, teacher retention at the school, teacher retention in the school district, teacher 
retention in the state, or teacher retention in the profession. 


BOX 1. HOW THE WWC REVIEWS AND DESCRIBES EVIDENCE 


The WWC evaluates evidence based on the quality and results of reviewed studies. The criteria the WWC uses for evaluating 
evidence are defined in the Procedures and Standards Handbooks and the Review Protocols. The studies summarized in this report 
were reviewed under WWC Standards (version 4.0) and the Teacher Excellence topic area protocol (version 4.0). 


To determine the effectiveness rating, the WWC considers what methods each study used, the direction of the effects, and the 
number of studies that tested the intervention. The higher the effectiveness rating, the more certain the WWC is about the reported 
results and about what will happen if the same intervention is implemented again. The following key explains the relationship between 


effectiveness ratings and the statements used in this report: 


Effectiveness Rating Rating interpretation 


Description of the evidence 


Positive (or negative) effects The intervention is likely to change an Strong evidence of a positive effect, with no 


outcome 


overriding contrary evidence 


Potentially positive (or negative) effects The intervention may change an outcome Evidence of a positive effect with no overriding 


contrary evidence 


No discernible effects The intervention may result in little to no No affirmative evidence of effects 


change in an outcome 


Mixed effects The intervention has inconsistent effects Evidence includes studies in at least two of 


onan outcome 


How is eMINTS Implemented? 


The following section provides details of how eMINTS was 
implemented. This information can help educators identify 
the requirements for implementing eM/NTS and determine 
whether implementing this intervention would be feasible 
in their district or school. Information on eMINTS presented 
in this section comes from the studies that meet WWC 
standards (Brandt et al., 2013; Martin et al., 2009) and from 
correspondence with the developer. 


Goal: eMINTS aims to help teachers improve student 
outcomes by integrating technology into their curriculum, 
implementing high-quality lesson plans, and building a 
learning community within their school. 


Target population: eM/NTS is offered to teachers of 
students in kindergarten through grade 12. 


Method of delivery: Teachers receive training in eMINTS 
through professional development offered online or in 
person and through coaching visits in their classrooms. 
Additional teaching resources are available through an 
online eMINTS portal. 


these categories: studies with positive effects, 
studies with negative effects, or more studies 
with indeterminate effects than with positive or 
negative effects 


Comparison group: In the two studies that contribute 
to this intervention report, students in the comparison 
group were taught by teachers who did not receive 


eMINTS training. Teachers may have received other 
training and professional development offered by their 
schools or school districts. 


Frequency and duration of service: Over the course of 
2 or more years, teachers participate in approximately 
140 hours of professional development through a series 
of sequenced learning sessions. Instructional specialists 
provide as many as 10 coaching sessions for teachers in 
their classrooms, with each session lasting up to 2 hours. 
Refer to Table 2 for additional details. 


Intervention components: The key components of 
eMINTS are described in Table 2. A shorter version of 

the eMINTS program called eMINTS4AII includes fewer 
professional development sessions and coaching visits. The 
main findings in this report measured the effectiveness of 
the eMINTS Comprehensive Program. 


N 


Table 2. Components of eMINTS 


Key component Description 


Sequenced 
professional 
development 


Over the course of 2 or more years, teachers participate in approximately 140 hours of professional development focused on 
integrating technology into the classroom and developing lessons that use inquiry-based learning, in which students engage 
in open-ended and meaningful investigations. Session topics also support building a community of learning in schools, 
student assessment, and classroom management. Sessions are offered in person or online and are held over 3 to 6 hours. 
The studies of eMINTS summarized in this review each provided more than 200 hours of professional development, rather 
than the recommended 140 hours. 


Coaching Instructional specialists provide up to 10 coaching sessions in teachers’ classrooms over 2 or more years. Each session is 
offered one on one and lasts from 1 to 2 hours. Specialists also offer coaching to groups of teachers to support collaboration 


and building a learning community within a school. In one of the studies of eM/NTS summarized in this review (Martin et al., 


2009), teachers received at least 20 coaching sessions over 2 years. 


Technology integration Instructional specialists help eMINTS classrooms integrate technology as a tool to support inquiry-based instruction, 
differentiated learning, and student assessment. Technology integration can take various forms. Examples of integration 
include computerized student assessments, or a teacher leading instruction from a computer blackboard while students 


participate at individual workstations. 


Online resources 


Teachers can access a range of materials through eMINTS’ online portal, including formative and summative assessments, 


teacher lesson plans and guides, and examples of high-quality classroom websites. The eMINTS program also provides 
online courses for teachers to supplement the professional development and coaching. At the end of the first year, teachers 
develop a classroom website with the help of the instructional specialist as a resource for students and their families (for 
example, to provide class schedules, homework assignments, and sample assessments). 


What Does eMINTS Cost? 


This preliminary list of costs is not designed to be exhaustive; 
rather, it provides educators an overview of the major resources 
needed to implement eM/NTS. The program costs described 
below are based on the information available as of July 2019. 


¢ Equipment and materials costs: To support technology 
integration, each student must have individual access to 
an Internet-connected computer or tablet. For schools that 
lack this access in the classroom or in a computer lab, the 
cost of providing computers is approximately $500 per 
student. The program can be adapted to the existing tech- 
nology in the school. 


¢ Personnel costs: For a fee of $10,000 per teacher for 
each year of the two-year program, eMINTS will provide 
instructional specialists to lead professional develop- 
ment and coaching activities. The developer may offer 
a discount on this fee when providing a custom quote to 
a district that meets certain criteria. Each participating 
classroom teacher is offered approximately 140 hours of 
professional development in sessions that are up to 6 hours 
long. In addition, teachers participate in as many as 10 
coaching sessions that are 1 or 2 hours long. Schools may 
choose to schedule professional development and coaching 
sessions within or outside of the school day. 


For More Information: 
About eMINTS 


eMINTS National Center, 111 Townsend Hall, Columbia, MO 65211 


Facilities costs: Professional development and coaching 
activities are offered online or in person at the school. For 
activities that occur in person, the school or district will 
need to supply space to host the teacher training. 


Costs paid by students or parents: There is no charge to 
students or parents. 


In-kind supports: To help offset the costs of eMINTS, 
schools that meet certain criteria may be eligible to receive 
discounts on educational products from eM/NTS partners. 


Sources of funding: The developer has provided eMINTS 
through various U.S. Department of Education grant 
programs, including the Student Support and Academic 
Enrichment program, the Investing in Innovation Fund, 
the Supporting Effective Educator Development grant 
program, and the Education Innovation and Research 
program. In Missouri, where the program was developed, 
grant funds from the Missouri Department of Elementary 
and Secondary Education have supported the program. 


Email: emints-info@emints.org. Web: http://emints.org/ and https://portal.emints.org/. Phone: (573) 884-7202 


About the cost of eM/INTS 


Information about eMI/NTS’ costs was provided by the developer. 


Research Summary 


The WWC identified 11 studies that investigated the 
effectiveness of eMINTS (Figure 1): 


* Two studies meet WWC group design standards with 
reservations 


¢ Six studies do not meet WWC group design standards 
¢ Three studies are ineligible for review 


The WWC reviews findings on the intervention’s effects on 
eligible outcome domains from studies that meet standards, 
either with or without reservations. Based on this review, the 
WWC generates an effectiveness rating, which summarizes 
how the intervention impacts, or changes, a particular outcome 
domain. The WWC reports additional supplemental findings, 
such as those the study authors reported as intermediate out- 
comes that reflect partial exposure to the intervention, on the 


Figure 1. Effectiveness ratings for eMINTS 


studies meets WWC 
jr laverstae lm tan) 
reservations 


0 studies meet WWC 


standards without 
reservations 


Contribute to effectiveness ratings 


WWC website (https://whatworks.ed.gov). These supplemen- 
tal findings and findings from studies that either do not meet 
WWC standards or are ineligible for review do not contribute 
to the effectiveness ratings. 


The two studies of eM/NTS that meet WWC group design 
standards reported findings on general mathematics achieve- 
ment and general literacy achievement. No other findings 

in the studies met WWC group design standards within any 
outcome domain included in the Teacher Excellence topic 
area.* Citations for the eight studies reviewed for this report 
are listed in the References section, which begins on page 10. 
Citations for the three studies that are ineligible for review 
and the reasons the WWC determined they were ineligible 
are also listed in the References section. 


6 studies do not 3 studies are 


meet WWC 
standards 


ineligible for 
review 


Do not contribute to effectiveness ratings 


eMINTS has potentially positive effects on general mathematics achievement 


Two studies that meet WWC group design standards with reservations contribute findings in the general mathematics 
achievement domain. The WWC determined one study showed evidence of a positive and statistically significant effect of 
eMINTS on general mathematics achievement (Brandt et al., 2013), and another study showed evidence of an 


indeterminate effect (Martin et al., 2009). 


eMINTS has no discernible effects on general literacy achievement 


Two studies that meet WWC group design standards with reservations contribute findings in the general literacy 
achievement domain. Both studies showed indeterminate effects (Brandt et al., 2013; Martin et al., 2009). 


Main Findings 


Table 3 shows the findings from two eMINTS studies that is potentially positive effects, indicating evidence of a positive 
meet WWC standards. The table includes WWC calculations effect with no overriding contrary evidence. These findings 
of the mean difference, effect size, and performance of the are based on 3,096 students. The effectiveness rating for 
intervention group relative to the comparison group. Based general literacy achievement is no discernible effects. These 
on findings from the two studies that meet WWC standards, findings are based on 3,245 students. 


the effectiveness rating for general mathematics achievement 


Table 3. Findings by outcome domain from studies of eMINTS that meet WWC standards 


Mean 
(standard deviation) WWC calculations 
Sample Intervention Comparison Mean Effect § Improvement 
Measure (study) Study sample size group group difference __size index p-value 
Missouri Assessment 7th and 8th 1,931 0.08 -0.05 0.13 0.13 +5 04 
Program: grade students (nr) (nr) 
Mathematics test 
(Brandt et al., 2013) 
Missouri Assessment 4th grade 640 648.36 651.17 -2.81 -0.09 4 109 
Program: students (30.03) (30.72) 
Mathematics test 
(Martin et al., 2009)’ 
Missouri Assessment 5th grade 525 670.22 669.67 0.55 0.02 +{ 93 
Program: students (33.71) (36.12) 
Mathematics test 
(Martin et al., 2009)? 
Outcome average for general mathematics achievement across all studies 0.04 +2 
Missouri Assessment 7th and 8th 2,079 0.05 0.09 -0.04 -0.04 -1 48 
Program: grade students (nr) (nr) 
Communication arts 
test (Brandt et al., 
2013)? 
Missouri Assessment 4th grade 640 659.29 663.89 -4.60 -0.15 6 134 
Program: students (31.12) (30.53) 
Communication arts 
test (Martin et al., 
2009)’ 
Missouri Assessment 5th grade 526 680.51 674.34 6.16 0.20 +8 25 
Program: students (30.09) (31.64) 
Communication arts 
test (Martin et al., 
2009)° 
Outcome average for general literacy achievement across all studies -0.01 0 


Notes: For mean difference and effect size values reported in the table, a positive number favors the intervention group and a negative number favors the comparison group. 
The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are given the 
intervention (measured in standard deviations of the outcome measure). An indicator of the effect of the intervention, the improvement index can be interpreted as the expected 
change in percentile rank for an average comparison group student if that student had received the intervention. For example, an improvement index of +2 means that the 
expected percentile rank of the average comparison group student would increase by 2 points if the student were placed in an eMINTS classroom. A positive improvement index 
does not necessarily mean the estimated effect is statistically significant. Some statistics may not sum as expected due to rounding. nr = not reported. 


@ Brandt et al. (2013) did not require corrections for clustering or multiple comparisons nor difference-in-differences adjustments. The study-reported means are standardized 
z-scores that combine the 7th and 8th grade tests and are adjusted for baseline student achievement and student, teacher, and school characteristics. This study is 
characterized as having a statistically significant positive effect on general mathematics achievement. The study is characterized as having indeterminate effects on general 
literacy achievement because the mean effect reported is not statistically significant. For more information, please refer to the WWC Procedures Handbook, version 4.0, page 22. 


> Martin et al. (2009) required corrections for clustering and multiple comparisons as well as a difference-in-differences adjustment. The p-values presented here were calculated 
by the WWC because the p-values reported in the study did not account for the clustered assignment of students to condition by classroom. Because the study did not report 
the number of clusters, the WWC calculated statistical significance using an assumption of at least 15 students per teacher in order to apply the WWC clustering correction, 
which led to no findings being statistically significant. The study reported the findings for the 4th and 5th grade communication arts tests as statistically significant, but they 

are not statistically significant after applying the WWC clustering correction. The finding for the 4th grade communication arts test is based on a regression adjustment, so a 
difference-in-differences adjustment was not needed. The WWC applied difference-in-differences adjustments to the unadjusted intervention group means and mean differences 
for the 5th grade communication arts test, and 4th and 5th grade mathematics tests. The WWC did not report the regression-adjusted results reported in the study for the 

5th grade test because the regression adjusted for both the baseline 3rd grade test score and the test score in 4th grade (an intermediate outcome). The magnitude of the 
regression-adjusted finding for the 4th grade mathematics test was not reported in the study. The sample sizes and unadjusted intervention and comparison group means were 
provided in response to an author query. This study is characterized as having indeterminate effects on general mathematics achievement and general literacy achievement 
because the mean effects reported are not statistically significant. For more information, please refer to the WWC Procedures Handbook, version 4.0, page 22. 


In What Context Was eMINTS Studied? 


The following section provides information on the setting of can help educators understand the context in which the studies 
the two studies of eMINTS that meet WWC standards, and a of eMINTS were conducted and determine whether the program 
description of the participants in the research. This information might be suitable for their setting. 


WHERE THE STUDIES WERE CONDUCTED 
2 studies, 3,245 students in as many as 74 public schools in Missouri. 
At least 39 of the schools were in rural areas 


Race Gender 


91% 


White Minority Female Male 


Free & Reduced-Price Lunch: 47% 


crs 1 2 3400 * OO 


Grades 4, 5, 7, 8 


Details of Each Study that Meets WWC Standards 


This section presents details for each study of eMINTS that Findings from Brandt et al. (2013) show evidence of a 
meets WWC standards. These details include the full study statistically significant positive effect of eMINTS in the 
reference, findings description, findings summary, and general mathematics achievement domain (Table 4). This 
description of study characteristics. A summary of domain finding is based on an outcome analysis that includes 1,931 
findings for each study is presented below, followed by a students. The finding on general literacy achievement, 
description of the study characteristics. These study-level which shows evidence of an indeterminate effect, is based 
details include contextual information about the study on an outcome analysis that includes 2,079 students. The 
setting, methods, sample, intervention group, comparison findings and research details summarized for this study 
group, outcomes, and implementation details. For additional come from four related citations, including the primary 
information, readers should refer to the original studies. study listed above. See the References section, which 


begins on page 10, for a list of all related publications. 
Research details for Brandt et al. (2013) 
Brandt, C., Meyers, C., & Molefe, A. (2013). The impact of 
eMINTS professional development on teacher instruction and 
student achievement: Year I report. Washington, DC: American 
Institutes for Research. 


Table 4. Summary of findings from Brandt et al. (2013) 


Meets WWC Group Design Standards With Reservations 


Study findings 
; Sample Average Improvement Statistically 
Outcome domain size effect size index significant 
General mathematics achievement 39 schools/1,931 students +0.13 +5 Yes 
General literacy achievement 39 schools/2,079 students -0.04 -1 No 


Table 5. Description of study characteristics for Brandt et al. (2013) 


WWC evidence 
rating 


Setting 


Methods 


Study sample 


Intervention 
group 


Comparison 
group 


Meets WWC Group Design Standards With Reservations. This is a cluster randomized controlled trial with low cluster- 
level attrition and individuals who are representative of the clusters.° For more information on how the WWC assigns study 
ratings, please see the WWC Procedures and Standards Handbooks (version 4.0) and WWC Standards Briefs, available on 
the WWC website. 


The study was conducted in 39 rural, high-poverty schools in Missouri in grades 7 and 8. Eligible schools were in rural areas 
and were either Title | schools or had a majority of students who were eligible for the free- or reduced-price meal program. 


The study used a cluster randomized controlled trial. The researchers randomly assigned 20 schools to each of three 
conditions in fall 2010: (1) the eMINTS program for 2 years and no further intervention in a third year, (2) eMINTS for 2 
years and a supplemental intervention (/ntel Teach) in a third year, or (3) a business-as-usual condition for all 3 years. The 
study randomly assigned the schools separately in three groups based on the grade ranges offered by the schools: schools 
with grades from pre-kindergarten to grade 8, schools with grades 5 to 8, and schools with grades 6 to 12. The analytic 
samples include students who entered schools after random assignment, and this review considered these students to pose 
a risk of bias. This study was therefore not eligible to be rated meets WWC group design standards without reservations, 
and the findings in the review potentially represent a combination of (1) the effect of the intervention on students and 

(2) a composition effect due to different types of students entering intervention and comparison schools. 


The main findings for this review compare outcomes for two groups: (1) students in eMINTS classrooms for up to 2 years who 
did not receive the supplemental intervention and (2) outcomes for students in the business-as-usual comparison condition. 
The eMINTS intervention was offered beginning in fall 2011, a year after random assignment, and concluded in fall 2013. The 
outcomes were measured in spring 2014, at the end of the first year after teachers completed their training in eMINTS. 


One school in the business-as-usual comparison condition closed during the study period, and students in that school did 
not contribute outcomes to the analytic sample. For both outcome measures, the sample loss of one school after random 
assignment (attrition) was within the acceptable threshold for the review: overall attrition was 2.5%, and differential attrition 
was 5 percentage points. Also for both outcome measures, the students included in the analytic sample were representative 
of all students in those grades present in the schools at follow-up: overall, 8.4% of the students present in the schools 

at follow-up were not included in the analytic sample, and the difference in non-response between the intervention and 
comparison schools was 2.2 percentage points. 


The 39 schools in the analytic sample included at least 2,079 students in grades 7 and 8. About 58% of students were 
eligible for free or reduced-priced lunch, less than 1% were limited English proficient, and 12% had an individual education 
plan. Fifty-one percent were male and 95% were White. 


Over 2 school years, teachers in the intervention group were offered the eMINTS program with approximately 240 hours 
of online and in-person professional learning and support, which included 46 sessions of sequenced professional learning 
and approximately 10 in-classroom coaching and mentoring visits. Teachers had access to additional online and in-person 
learning opportunities and resources designed to support high-quality instruction and student assessment. Teachers 
worked individually and collaboratively with eMINTS instructional specialists and others in the training to complete learning 
activities, such as creating lesson plans and a class website. Additionally, a technology coordinator in each school was 
trained to support technology integration in eMINTS teachers’ classrooms. 


Students in the comparison group were taught by teachers who did not receive the eMINTS program. Teachers may have 
received other business-as-usual training and professional development offered by their schools or school districts. 


Outcomes and 
measurement 


Additional 
implementation 
details 


Study authors reported findings on two outcome measures in student achievement that are eligible for review under the 
Teacher Excellence topic area and meet WWC group design standards. These outcomes were measured by the Missouri 
Assessment Program mathematics and communication arts tests in spring 2012, 2013, and 2014. The outcomes measured 
in spring 2014 are considered the main findings for this review because they are measured in the first year after teachers 
completed their training in eMINTS, which occurred from fall 2011 to spring 2013. The findings in other years are considered 
supplemental findings and are reported on the WWC website (https:/Awhatworks.ed.gov). The supplemental findings do not 
factor into the intervention’s rating of effectiveness. 


Two additional student outcomes were collected but did not meet WWC group design standards. The 21st Century 

Skills test was used to measure students’ creativity and innovation, problem solving, decision making, and comfort with 
technology. A student survey was used to measure student engagement in the school. The findings for these measures 
did not meet WWC group design standards because they included outcomes for students who entered study schools after 
random assignment. This review considered these students to pose a risk of bias, and the WWC was unable to assess 
either the representativeness of the analytic samples of students or the equivalence of the analytic intervention and 
comparison groups at baseline. 


The study also measured teacher outcomes related to instructional practice using teacher surveys and classroom 
observations. The teacher survey measures and the teacher technology use measure based on classroom observations 
were not eligible for review in the instructional practice domain. Eligible measures in this domain must have a statistical 
relationship with a student outcome (such as student achievement) that is documented by the study authors or cited in 
another publication, but this information was not available for these measures. Other classroom observation measures 
were assessed using the Classroom Assessment Scoring System-Secondary. Findings for these outcomes did not meet 
WWC group design standards because they included outcomes for teachers who entered study schools after random 
assignment. This review considered these teachers to pose a risk of bias, and the WWC was unable to assess either the 
representativeness of the analytic samples of teachers or the equivalence of the analytic intervention and comparison 
groups at baseline. 


The study also reported findings for student and teacher outcomes for schools that provided the eMINTS program for 2 
years followed by the supplemental Intel Teach intervention for 1 year. Intel Teach was offered to further increase teachers’ 
technology integration skills in the classroom and involved 42 hours of professional development in person and online, 
four to five coaching sessions, and access to online resources. Because the eMINTS intervention was bundled with /ntel 
Teach for this supplemental intervention condition, the findings were not eligible for review and do not contribute to the 
effectiveness rating for eMINTS. 


The eMINTS professional development was facilitated by trained eMINTS staff supported by staff at the eMINTS National 
Center who monitored and supervised program delivery. Study authors conducted a study of implementation and found that 
the professional development and essential technology resources and guidance needed to support the eM/NTS program in 
study schools were delivered as designed. 


Research details for Martin et al. (2009) Findings from Martin et al. (2009) show evidence of indeter- 
Martin, W., Strother, S., & Reitzes, T. (2009). eMINTS 2009 minate effects of eM/NTS on general mathematics achieve- 


program evaluation report: An analysis of the persistence of 
program impact on student achievement. New York: Center 


ment and general literacy achievement (Table 6). These 
findings are based on outcome analyses that include 1,165 


for Children & Technology. Retrieved from http://emints.org/ and 1,166 students in each domain, respectively. 
wp-content/uploads/2014/11/eMINTS-2009-Program-Evalua- 


tion.pdf. 


Table 6. Summary of findings from Martin et al. (2009) 


Meets WWC Group Design Standards With Reservations 


Study findings 
Sample Average Improvement Statistically 
Outcome domain size effect size index significant 
General mathematics achievement 1,165 students -0.04 -2 No 
General literacy achievement 1,166 students +0.02 +{ No 


Table 7. Description of study characteristics for Martin et al. (2009) 


WWC evidence 
rating 


Setting 


Methods 


Study sample 


Intervention 
group 


Comparison 
group 


Outcomes and 
measurement 


Additional 
implementation 
details 


Meets WWC Group Design Standards With Reservations. This is a cluster quasi-experimental design study that satisfies 
the baseline equivalence requirement for the individuals in the analytic intervention and comparison groups.® 


The study took place in 35 schools across 10 school districts in Missouri. 


The study used a cluster quasi-experimental design and compared outcomes for two groups: (1) students in classrooms with 
teachers who were trained in eMINTS and (2) students in classrooms without eM/NTS-trained teachers. The comparison 
group classrooms were mostly those in the same grade and school as eMINTS classrooms. If that was not possible because 
all teachers in a school and grade received eMINTS training, the researchers instead included classrooms with teachers 
who did not receive the intervention who were in the same grade and within the same school district, but in another school 
that the researchers considered comparable. From within the intervention classrooms, the researchers included students in 
the analytic sample who had been taught by an eMI/NTS-trained teacher in 2 consecutive years. Similarly, from within the 
comparison group classrooms, the researchers included students in the analytic sample who had been taught by a teacher 
not trained in eMINTS in 2 consecutive years. 


The study findings that meet WWC group design standards include students in 4th and 5th grades. The 5th grade students 
were exposed to eMINTS in both 4th and 5th grades. Although the 4th grade students were exposed to eMINTS in both 3rd 
and 4th grades, the findings for the grade 4 sample reflect just a single year of exposure to the intervention because the 
analysis adjusts for baseline achievement measured at the end of grade 3. 


The study sample in 4th grade included 640 students (328 and 312 in the intervention and comparison groups, 
respectively). The study sample in 5th grade included 526 students (206 and 320 in the intervention and comparison 
groups, respectively). The authors do not report the number of teachers and classrooms in the study. Forty-seven percent 
of the students were female, 16% were students the authors described as belonging to a minority group, and 28% were 
eligible for the free- or reduced-price meal program. 


Over 2 school years, teachers in the intervention group were offered the eMINTS program with 250 hours of professional 
development and support, including 10 to 12 classroom coaching visits each year by eMINTS instructional specialists. 
Technology coordinators in each school were trained by eMINTS to support teachers’ integration of technology in the classroom. 


Students in the comparison group were taught by teachers who did not receive training by eMINTS. Teachers may have 
received other business-as-usual training and professional development offered by their schools or school districts. 


Study authors reported findings on two student outcome measures that are eligible for review under the Teacher Excellence 
topic area. These are the Missouri Assessment Program tests of mathematics and communication arts skills in grades 

4 through 6. These same tests administered in grade 3 are used as baseline measures for analyses of grade 4 and 5 
outcomes, while the grade 4 test is used as the baseline measure for grade 6. Findings for these tests given in grades 4 
and 5 meet WWC group design standards with reservations. However, findings for the tests given in grade 6 do not meet 
WWC group design standards because the baseline equivalence requirement is not satisfied. 


The eMINTS professional development was implemented by eMINTS instructional specialists. The study authors conducted 
an implementation study and found that eMINTS was implemented with a high degree of fidelity based on observations of 
professional development sessions, records of classroom visits, and reviews of teacher artifacts. 
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Endnotes 


' Hanushek (2011) and Chetty et al. (2014), for example, describe 
differences across teachers in their impacts on academic achieve- 
ment and earnings. 


> The descriptive information for this intervention comes from the 
studies that contribute to this intervention report and the interven- 
tion developer. The What Works Clearinghouse (WWC) requests 
developers review the intervention description sections for 
accuracy from their perspective. The WWC provided the devel- 
oper with the intervention description in July 2019 and the WWC 
incorporated feedback from the developer. Further verification of 
the accuracy of the descriptive information for this intervention is 
beyond the scope of this review. 


>The literature search reflects documents publicly available by June 
2019. Reviews of the studies in this report used the standards from 
the WWC Procedures and Standards Handbooks (version 4.0) and 
the Teacher Excellence review protocol (version 4.0). The evidence 
presented in this report is based on available research. Findings 
and conclusions could change as new research becomes available. 


‘The effects of eMINTS are not known for other outcome domains 
within the Teacher Excellence topic area, including general science 
achievement, general social studies achievement, general achieve- 
ment, English language proficiency, staying in school, progression 
in school, completing school, student social interaction, observed 
individual behavior, student emotional status, student engagement 
in school, instructional practice, teacher attendance, teacher reten- 
tion at the school, teacher retention in the school district, teacher 
retention in the state, and teacher retention in the profession. 


> Because Brandt et al. (2013) included joiners who pose a risk of 
bias but had low cluster-level attrition, to meet WWC group design 
standards, the study must either demonstrate that the samples of 
students contributing outcome data to the analytic samples were 
representative of all students in those schools and grades or estab- 
lish baseline equivalence of the analytic intervention and compar- 
ison groups. The study meets WWC group design standards with 
reservations because it establishes that the analytic samples are 
representative. The WWC assesses how representative the analytic 
sample is by comparing the number of students present in the 
analytic sample of study schools at follow-up who do not contrib- 
ute outcome data to the total number of students in those schools 
(that is, the non-response rate). This proportion overall and the 
difference in non-response between the intervention and compari- 
son group is compared to the published WWC attrition table using 
the attrition boundary specified in the review protocol, which is the 
optimistic boundary for the Teacher Excellence review protocol 
(see the WWC Procedures and Standards Handbooks, version 4.0, 
available on the WWC website). This review of Brandt et al. (2013) 
and the rating decision differ from the prior review of the study 
conducted for the Institute of Education Sciences (IES) grant com- 
petition (using the Version 3.0 review standards and the Teacher 
Training, Evaluation, and Compensation Review Protocol, Version 
3.1). The prior review rated the study Meets WWC Group Design 
Standards Without Reservations. The difference in rating decisions 
is due to the updated review standards for cluster-design studies. 


® Martin et al. (2009) is a quasi-experimental design study that satis- 
fies the baseline equivalence requirement by demonstrating equiv- 
alence of the intervention and comparison groups on the math and 
literacy outcome measures for the 4th and Sth grades. Thus, the 
study meets WWC group design standards with reservations. This 
review of Martin et al. (2009) and the rating decision differ from 
the prior review of the study conducted for the IES grant competi- 
tion (using the Version 3.0 review standards and the Single Study 
Review Protocol, Version 2.0). The prior review rated the study 
Does Not Meet WWC Group Design Standards. The difference in 
rating decisions is because the prior review considered differences 
in student economic disadvantage (the percentage eligible for the 
free- or reduced-price meal program) between the intervention 
and comparison group as evidence that the study does not satisfy 
the baseline equivalence requirement. The current review protocol 
only requires that the study satisfy the baseline equivalence re- 
quirement on the pre-intervention measure of the outcome. 
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